System, Method and Apparatus for Sensing Changes in an Environment Using Wireless Communication Signals

ABSTRACT

A wireless signal-based sensing system is provided. The system includes a plurality of devices, each device capable of sending and receiving wireless signals to create a sensing area. The system also includes at least one connection mechanism to enable at least one of the devices to connect to an application, and at least one analytics application for processing measurements of wireless signals obtained from the sensing area.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.16/002,944 filed on Jun. 7, 2018, which is a continuation of PCTApplication No. PCT/CA2016/051533 filed on Dec. 22, 2016, which claimspriority to U.S. Provisional Patent Application No. 62/387,174 filed onDec. 23, 2016, all incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to object detection, localization,tracking and activity recognition within an area of interest for sensingchanges in an environment using wireless communication signals.

DESCRIPTION OF THE RELATED ART

Many forms of object detection, motion detection and activityrecognition exist today, including optical and thermal/infrared cameras,passive/active infrared motion detectors, acoustic sensors, vibrationsensors, cameras, induction coils, and radio frequency (RF) sensors.These technologies can be useful in applications such as security, homeautomation, elderly and child monitoring, and others.

One of several challenges of existing object detection, motion detectionand activity recognition technologies is the requirement to deployadditional network infrastructure in order to support sensorcommunication.

Recent research and advancements have developed sensing techniques thatutilize measurements available through state monitoring of existingwireless systems and devices currently used only for communicationpurposes.

SUMMARY

The following relates to the creation of a sensing area for activityrecognition by re-using particular information, e.g., informationavailable in the lower layers of the OSI reference model of existingwireless communication systems. Systems, methods and apparatus areprovided in order to create a wireless signal-based sensing platformthat employs local and/or remote processing capabilities for objectdetection, localization, tracking and activity recognition.

The following also proposes a system, method, and apparatus that cancollect fine-grained measurements available in existing wireless systemsand devices that can be used for activity recognition withoutnecessitating the addition of new network infrastructure as currentlyrequired. An example of these fine-grained measurements is the channelstate information (CSI) measurements in systems such Wi-Fi and regulatedby the IEEE 802.11n and IEEE 802.11ac standards, which providecontinuous fine-grained measurements characterizing the behavior of thewireless channel between a transmitter and a receiver.

In one aspect, there is provided a wireless signal-based sensing systemcomprising: at least one sensing area generated by a plurality ofdevices, each device in the sensing area capable of sending andreceiving wireless signals according to a communication protocol,wherein the communication protocol comprises at least one existingmechanism at a first layer of the devices for sensing a communicationchannel between pairs of connected devices in the sensing area; at leastone application of at least one of the plurality of devices to access atleast the first layer of the device to obtain measurements sensed by thecommunication protocol using the existing mechanism, wherein the atleast one application is configured to generate traffic on thecommunication channel when an insufficient amount of network traffic ispresent; and at least one analytics application for receiving andprocessing measurements of wireless signals obtained from the sensingarea by the plurality of devices.

In another aspect, there is provided a method for wireless signal-basedsensing comprising having a sensing area generated by a plurality ofdevices, each device in the sensing area capable of sending andreceiving wireless signals according to a communication protocol,wherein the communication protocol comprises at least one existingmechanism at a first layer of the devices for sensing a communicationchannel between pairs of connected devices in the sensing area;establishing the communication channel to generate sensed data at thefirst layer; enabling at least one application of at least one of theplurality of devices to access at least the first layer of the device toobtain measurements sensed by the communication protocol using theexisting mechanism, wherein the at least one application is configuredto generate traffic on the communication channel when an insufficientamount of network traffic is present; and receiving and processingmeasurements of wireless signals by at least one analytics application,the measurements having been obtained from the sensing area by theplurality of devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of example only with referenceto the appended drawings wherein:

FIG. 1(a) illustrates a system able to sense objects within an area viawireless signals;

FIG. 1(b) illustrates data extraction functionalities of the devicewherein relevant measurements are taken from various layers of the OSIreference model;

FIG. 2 illustrates a system able to sense objects within an area viawireless signals by connecting at least two instances of the device;

FIG. 3 illustrates a generic physical embodiment of the device;

FIG. 4(a) illustrates an example of the functional logic of a devicewhen operating on inputs from the wireless sensing system;

FIG. 4(b) illustrates an example of the functional logic of the devicewhen operating on inputs received from an external device or user;

FIG. 5 illustrates an example of the functional logic of an analyticsapplication which could be run, in whole or in part, across multipledevices, including the device in the above figures, and/or one or moreremote devices; and

FIG. 6 illustrates an example of the system proposed herein by using aW-Fi network in which one of the devices is able to collect channelstate information (CSI) and send raw data and/or pre-processed data to acloud-based application.

DETAILED DESCRIPTION

As illustrated in FIG. 1(a) an active sensing area 100 is generated viaa wireless communication between a device 102 and a communicationnetwork 104, which contains at least one node 108 able to transmit andreceive wireless signals to and from device 102. Device 102 could be anew device or an existing device modified in order to be capable ofextracting measurements required for sensing. The active sensing area100 includes at least two connected wireless nodes 108 exchanginginformation about the states of the wireless signals and/or channels. InFIG. 1(a), the device 102 can be considered the second node within theactive sensing area 100 that is illustrated.

Most current wireless communication devices implement internalmechanisms for sensing wireless channel states in order to maximizechannel capacity and communication robustness. For example, if the opensystems interconnection (OSI) reference model is used, then in order togenerate relevant measurements for the purposes of activity recognitionthrough the wireless signal-based sensing system proposed herein, thedevice 102 should connect to at least one other wireless node 108 withinthe existing communication system with similar physical layercharacteristics. The information that is relevant for activityrecognition usually remains in the lower layers of the OSI model. Theselayers are usually the physical layer, data link layer and/or networklayer.

One of the functionalities of the device 102 is to collect measurementsfrom the lower layers of the OSI model, as shown in FIG. 1(b), andencapsulate the measurements as data, according to the specifications ofthe communication system employed for sending the data to an analyticsapplication 110. The analytics application 110 could be hosted withinthe local network or in a remote network following a cloud-basedarchitecture as shown in FIG. 1(a). The analytics application 110 caninteract with external systems or applications which are also “in thecloud” through, for example, an application programming interface (API).The network 104 comprises at least one node 108 provided with thehardware and logic units needed for interconnecting the communicationnetwork 104 with at least one remote network where the cloud system 106hosts the application 110. As was mentioned, the wireless connectionbetween the device 102 and at least one of the nodes within thecommunication network 104 generates the active sensing area 100. Thedevice 102 becomes one node of the communication network 104. It isworth noting that device 102 could be an additional device added to anexisting network 104, or it could be a device already included innetwork 104 but modified in order to collect the measurements describedherein.

One or more of the plurality of devices 102 can include a mechanism toremain fixed in three-dimensional space in order to ensure consistencyof measured changes in the environment. Such a mechanism can be used toaddress the fact that the W-Fi devices should remain fixed or themeasurements of the attenuation, and phase shifts due to changes in thereflections, obstructions, scattering, among other propagationmechanisms, of the travelling wireless signals (and hence the baselinemeasurements of the environment) will change. If baseline measurementschange, the system would need to re-characterize (i.e., train) for thenew device position or compensate according to the new baseline. Byfixing the device 102, less training and/or processing is required forit to become useful in the first place, as well as thereafter.

In one of the embodiments described herein a communication network 200comprises at least two devices 102 as shown in FIG. 2. In thisembodiment, devices 102 comprise the entire communication network. Byemploying two instances of device 102, referred to as Device 1 andDevice 2, a sensing area 100 is created as illustrated in FIG. 2. Ifpart or all of the analytics application 110 is hosted in a remotefacility, at least one of Device 1 or Device 2 needs to be capable ofconnecting to the remote network where the application 110 is hosted. Ifadditional devices 102 are incorporated into the sensing system, theactive sensing area 100 is enhanced and/or extended according to thenumber and location of new devices available within the communicationnetwork 200 and their wireless communication range. Enhancement of thesensing area occurs as a result of the increase in the number of datasources available. Extension of the sensing area occurs as a result ofthe increase in overall reach of the wireless network 200. The scope ofthe systems and methods proposed herein are not limited by anyparticular network topology. The communication network 200 could becreated by following any of the regulated communication standards, e.g.IEEE 802.11 standard family or some new standard.

The basic functional blocks of device 102 are represented in FIG. 3. Thedevice 102 comprises components in order to enable one or more of thefunctionalities shown in FIG. 3 and/or data flow shown in FIGS. 4(a) and4(b). A power source 300 is one example of one of these components sinceit provides the required DC and/or AC voltage while supporting the powerconsumption of all the different functional blocks implemented in device102. A wireless transceiver module 302 comprises components needed forsending and receiving wireless signals, e.g. radiation system,amplifiers, filters, mixers, local oscillators, ADC and DAC, and anyother component required in the modulator and demodulator. The wirelesstransceivers module 302 is used to provide at least one wireless linkwith at least one node 108 within a network 104, or with another deviceof the same class, i.e. another device 102 as in FIG. 2. The transceivermodule 302 can include more than one wireless network interfacedepending on the functionalities the designer includes in the device. Anoptional wireless network interface 2 is represented in dashed lines aswell as an optional network interface 1 304, with its respective port.An optional network interface 3 could support a wireless link to aremote network, either a network of the same class or a different classto the one generating the sensing area 100, in case the local network isnot available due to a power failure or a malfunctioning of a componentof the local network. In case of a power failure, the device 102 couldbe capable of using a backup battery 314 to power a connection to asecondary network of the same or different class in order to forwarddata to or from a critical application.

In FIG. 3, the logic unit A is a program or multiple programs formanaging and controlling the functions associated with the transceiverscomprising both wired network interfaces as 304 or wireless networkinterfaces as within wireless transceivers 302 and with any otherfunctional block within the device that requires a program to implementits functionalities. The logic unit B is the program to interact withthe information that usually remains in the lower layers of the OSIreference model while controlling the communication between at least twowireless nodes. The lower layer of the OSI reference model we refer toare at least the physical layer and the data link layer as shown in FIG.1(b). An example of those measurements is the CSI measurements withinIEEE 802.11n and IEEE 802.11ac standards. The processing unit 308provides processing resources required to execute any of the programsand/or applications designed for a specific implementation of device102. The logic unit B in 306 could also include information not onlyabout the wireless channel but also about the current performance of thewireless transceivers.

If there is no network and at least two devices 102 are used to create asensing area 100 as in FIG. 2, at least one instance of device 102 needsto send data to an application regardless of where the application ishosted. This functionality is represented by 316 in FIG. 3, and enablesthe device 102 to transfer data from one network to another. Additionalfunctionalities can be introduced in the implementation of the device102 represented by block 318 in FIG. 3. Some examples include, withoutlimitation, input/output interfaces, a speaker, a siren, etc.

FIG. 4(a) is a flow chart illustrating an example of the functionallogic of a device 102 when operating on inputs from the wireless sensingsystem. The device 102 connects to a network or connects to anotherdevice 102 of the same class, as noted above. The device 102 thendetermines if there is sufficient wireless traffic allowing it toperform appropriate sensing (for instance by ensuring that enoughinformation and/or measurements within the lower layers of the OSI modelhave been generated for a robust sensing reading). If not, the device102 generates network traffic in order to allow and/or improve sensingeven though there may not be any requirement for the transfer ofwireless data over the network and/or devices. If there is sufficientwireless traffic, the device 102 obtains non-invasive access to lowerlayer data measurements, e.g., channel state information per receivedpackage accordingly to IEEE 802.11n, IEEE 802.1ac, and copies this lowerlayer data. The device 102 then determines if buffering is required and,if so, buffers the lower layer data. Similarly, the device 102 thendetermines if any formatting is required and, if so, formats the lowerlayer data.

The device 102 then determines if any local pre-processing is required.If no pre-processing is required, the device 102 can encapsulate outputdata to the connected network and thus send data to a remote application110. On the other hand, if preprocessing is required, a local analyticsapplication 310 implements the preprocessing of the measurements, e.g. alocal machine learning feature and/or a compression method forcompressing the formatted measurements and then sends the results out tothe remote application 110. The output from this local analyticsapplication 310 is encapsulated according to the requirements of theconnected network. The device 102 also determines if any action isrequired on connected actuators. If so, commands are sent to thoseconnected actuators in addition to sending the pre-processed data to theremote application 110. The connected actuators can be any externaldevice that moves or controls an external mechanism or system when thecontrol signal is received from the system proposed herein. If theactuators are not directly connected to device 102, the analyticsapplication 110 can interact with an external API developed andimplemented to control the actuators, which can be hosted in the cloud.As such, if the actuator is directly connected (e.g., through a WLAN, anEthernet connection, USB tethering, etc.), the output generated by theanalytics application 110 can be shared with the actuator by employingthe local connection. In case the actuator interacts through acloud-based system, the analytics application 110 in the cloud can shareits output(s) in the cloud system.

FIG. 4(b) illustrates the functional logic performed by the device 102when the control signal is generated in the analytics application 110 inthe cloud system 106, or when operating on inputs received from anexternal device or directly from a user. The device 102 operates toanswer an external request by determining if any action is required onconnected actuators. If so, commands are sent to the connected actuatorsaccordingly and a reply is sent to the external request. If not, thedevice 102 then determines if the external request includes newconfiguration parameters. If so, the new configuration is downloaded andloaded on the device 102 and a reply is sent to the external request. Ifnot, the device 102 determines if the external request requires reset.If so, the device 102 is reset and the device replies to the externalrequest. If the external request is not recognized, a message indicatingsuch is sent as a reply to the external request.

FIG. 5 illustrates an example of the functional logic of an analyticsapplication 110 which could be run, in whole or in part, across multipledevices 102. The device 102 obtains raw measurements and pre-processesthose raw measurements. The analytics application 110 then computesmachine learning features or operations and/or identifies relevantgroupings or measurement clusters. The analytics application 110 alsocollects and/or generates and/or infers external and/or internal labelsintended to classify the measurement data and/or features and/orclusters in a meaningful way.

Typically, unlabeled data includes samples of natural or human-createdartifacts that one can obtain from the world. Some examples of unlabeleddata might include photos, audio recordings, videos, news articles,tweets, x-rays, etc. There is no “explanation” for each piece ofunlabeled data—it just contains the data. Labeled data typically takes aset of unlabeled data and augments each piece of that unlabeled datawith some sort of meaningful “tag,” “label,” or “class” that is somehowinformative or desirable to know. For example, labels for the abovetypes of unlabeled data might be whether this photo contains an animalor human, which words were uttered in an audio recording, what type ofaction is being performed in this video, what the topic of this newsarticle is, etc. Labels for data are often obtained by asking humans tomake judgments about a given piece of unlabeled data. After obtaining alabeled dataset, machine learning models can be applied to the data sothat new unlabeled data can be presented to the model and a likely labelcan be guessed or predicted for that piece of unlabeled data.

The analytics application 110 then applies one or more core algorithmsbased on digital signal processing and machine learning techniques forrecognizing new instances of the identified clusters. That is, themachine learning techniques can discover and label clusters of data thatinfer some activity and then monitor new data to recognize similarclusters of data. In this way, the machine learning can infer that thesame activity is being performed. The analytics application 110 may thendetermine one or more appropriate output responses based on theprocessed measurements. For example, if the sensing system detected astranger lurking outside a window of a private residence, an appropriateresponse might be to alert the homeowner and/or local law enforcementwith a text message. In summary, the above process can include:Identification of clusters, labels collection (either provided by theusers or inferred by specific analytics applications), detection of theprevious identified and labelled clusters but now on fresh data comingin, and notification of the activity performed by using the appropriatelabel(s).

FIG. 6 illustrates an example of the system proposed herein by using aW-Fi network in which one of the devices is able to collect channelstate information (CSI) and send raw data and/or pre-processed data to acloud-based application. In the example shown in FIG. 6, the rawmeasurements collected in a device 102 correspond to channel stateinformation measurements of one stream (a link between one of theantennas in the transmitter and one of the antennas in the receiver) formultiple packages that device 102 received from the wireless accesspoint 600, The wireless access point 600 is capable of routing packagesto a remote network where the analytics application 110 is hosted. Themagnitude of the CSI measurements is used in 602 in order to representall the frequency components that were measured for each of the acquiredsamples, e.g. each sample could correspond with the CSI measurement foreach of the available subcarriers for each package that was received bydevice 102. In this example, the actual data encapsulated and sent viathe link 604 corresponds to the channel state information availablewithin a certain bandwidth, and for certain subcarriers as establishedand regulated in IEEE 802.11n and/or IEEE 802.11ac, and for all thestreams generating the sensing area 100.

For simplicity and clarity of illustration, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements. In addition, numerousspecific details are set forth in order to provide a thoroughunderstanding of the examples described herein. However, it will beunderstood by those of ordinary skill in the art that the examplesdescribed herein may be practiced without these specific details. Inother instances, well-known methods, procedures and components have notbeen described in detail so as not to obscure the examples describedherein. Also, the description is not to be considered as limiting thescope of the examples described herein.

It will be appreciated that the examples and corresponding diagrams usedherein are for illustrative purposes only. Different configurations andterminology can be used without departing from the principles expressedherein. For instance, components and modules can be added, deleted,modified, or arranged with differing connections without departing fromthese principles.

It will also be appreciated that any module or component exemplifiedherein that executes instructions may include or otherwise have accessto computer readable media such as storage media, computer storagemedia, or data storage devices (removable and/or non-removable) such as,for example, magnetic disks, optical disks, or tape. Computer storagemedia may include volatile and non-volatile, removable and non-removablemedia implemented in any method or technology for storage ofinformation, such as computer readable instructions, data structures,program modules, or other data. Examples of computer storage mediainclude RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by an application, module,or both. Any such computer storage media may be part of the system, anycomponent of or related thereto, or accessible or connectable thereto.Any application or module herein described may be implemented usingcomputer readable/executable instructions that may be stored orotherwise held by such computer readable media.

The steps or operations in the flow charts and diagrams described hereinare just for example. There may be many variations to these steps oroperations without departing from the principles discussed above. Forinstance, the steps may be performed in a differing order, or steps maybe added, deleted, or modified.

Although the above principles have been described with reference tocertain specific examples, various modifications thereof will beapparent to those skilled in the art as outlined in the appended claims.

1. A wireless signal-based sensing system comprising: at least onesensing area generated by a plurality of devices, each device in thesensing area capable of sending and receiving wireless signals accordingto a communication protocol, wherein the communication protocolcomprises at least one existing mechanism at a first layer of thedevices for sensing a communication channel between pairs of connecteddevices in the sensing area; at least one application of at least one ofthe plurality of devices to access at least the first layer of thedevice to obtain measurements sensed by the communication protocol usingthe existing mechanism, wherein the at least one application isconfigured to generate traffic on the communication channel when aninsufficient amount of network traffic is present; and at least oneanalytics application for receiving and processing measurements ofwireless signals obtained from the sensing area by the plurality ofdevices.
 2. The system of claim 1, wherein the first layer is a layer ofthe OSI reference model.
 3. The system of claim 1, wherein the at leastone analytics application is configured to process one or more of:temporal changes in wireless signal intensity, channel frequencyresponse, impulse response, or any other measurable variables of thewireless signals that are sensitive to changes in an environment.
 4. Thesystem of claim 1, wherein the at least one analytics application isconfigured to: create one or more clusters comprising relevant groupingsof measurements from the measured data in order to associate thoseclusters with one or more of: specific subjects, locations, movements oractivities; and store the identified clusters and collects and generateslabels for the clusters corresponding to one or more of: subjects,locations, movements or specific activities.
 5. The system of claim 4,wherein the at least one analytics application is configured to: applyat least one of digital signal processing and machine learning, torecognize one or more of: subjects, locations, movements or activitiesin real-time, based on the stored clusters.
 6. The system of claim 1wherein the at least one analytics application resides, in whole or inpart, on one or more of a remote computer, a local computer or one ormore of the devices.
 7. The system of claim 3, wherein at least one ofthe devices is configured to: read one or more measurable variables;store one or more measurable variables; and send one or more measurablevariables to at least one of another device and the analyticsapplication.
 8. The system of claim 7, wherein the at least oneanalytics application is configured to format the one or more measurablevariables.
 9. The system of claim 8, wherein the at least one analyticsapplication is configured to process the formatted measurements.
 10. Thesystem of claim 9, wherein the at least one analytics application isfurther configured to determine one or more output responses based onthe processed measurements.
 11. The system of claim 10, wherein at leastone of the devices is configured to send one or more of: processedmeasurements or output responses to one or more of another device or theanalytics application.
 12. The system of claim 1, wherein at least oneof the plurality of devices is capable of transferring data to theanalytics application using one or more network topology.
 13. The systemof claim 12, wherein the one or more network topology is defined in theIEEE 802.11 family standards for Wi-Fi communications.
 14. The system ofclaim 1, wherein at least one of the plurality of devices is capable oftransferring data to the analytics application using another wirelesscommunication standard.
 15. The system of claim 14, wherein the otherwireless communication standard is any one of: Zigbee, Bluetooth, 3G,4G, and LTE.
 16. The system of claim 1, wherein at least one devicecomprises: at least one processor; at least one wireless interface tointeract with one or more of a wireless network or another device; afirst logic unit capable of extracting measurable variables of thewireless signals which are sensitive to changes in the environment; asecond logic unit capable of formatting the measurement data and sharingit with a network and/or another device to which the device isconnected; computer executable instructions to initiate wirelesscommunications in order to interact with at least one other device inorder to generate the sensing area.
 17. The system of claim 16, whereinthe at least one device further comprises one or more of: a wirelesssignal repeater functionality in order to extend the coverage area ofthe wireless sensing network the device is associated to; circuitry tosupply, monitor and/or control power to one or more external devices; atleast one mechanism to send wireless communications to one or moreexternal devices for at least one of: control, notification or otherdata transmission; at least one mechanism to send wired communicationsto external devices for one or more of control, notification or anotherdata transmission.
 18. The system of claim 1, wherein at least one ofthe plurality of devices comprises a mechanism to remain fixed inthree-dimensional space in order to ensure consistency of measuredchanges in the environment.
 19. A method for wireless signal-basedsensing comprising: having a sensing area generated by a plurality ofdevices, each device in the sensing area capable of sending andreceiving wireless signals according to a communication protocol,wherein the communication protocol comprises at least one existingmechanism at a first layer of the devices for sensing a communicationchannel between pairs of connected devices in the sensing area;establishing the communication channel to generate sensed data at thefirst layer; enabling at least one application of at least one of theplurality of devices to access at least the first layer of the device toobtain measurements sensed by the communication protocol using theexisting mechanism, wherein the at least one application is configuredto generate traffic on the communication channel when an insufficientamount of network traffic is present; and receiving and processingmeasurements of wireless signals by at least one analytics application,the measurements having been obtained from the sensing area by theplurality of devices.
 20. (canceled)
 21. (canceled)
 22. (canceled) 23.(canceled)
 24. (canceled)
 25. (canceled)
 26. (canceled)
 27. (canceled)28. (canceled)
 29. (canceled)
 30. (canceled)
 31. (canceled) 32.(canceled)
 33. (canceled)
 34. (canceled)
 35. (canceled)
 36. (canceled)37. A computer readable medium comprising computer executableinstructions for wireless signal-based sensing comprising instructionsfor: having a sensing area generated by a plurality of devices, eachdevice in the sensing area capable of sending and receiving wirelesssignals according to a communication protocol, wherein the communicationprotocol comprises at least one existing mechanism at a first layer ofthe devices for sensing a communication channel between pairs ofconnected devices in the sensing area; establishing the communicationchannel to generate sensed data at the first layer; enabling at leastone application of at least one of the plurality of devices to access atleast the first layer of the device to obtain measurements sensed by thecommunication protocol using the existing mechanism, wherein the atleast one application is configured to generate traffic on thecommunication channel when an insufficient amount of network traffic ispresent; and receiving and processing measurements of wireless signalsby at least one analytics application, the measurements having beenobtained from the sensing area by the plurality of devices.